Toward a Scoring Function for Quality-Driven Machine Translation

We describe how we constructed an automatic scoring function for machine translation quality; this function makes use of arbitrarily many pieces of natural language processing software that has been designed to process English language text. By machine-learning values of fnnctions available inside the software and by constructing functions that yield values based upon the software output, we are able to achieve preliminary, positive results in machine-learning the difference between human-produced English and machine-translation English. We suggest how the scoring ftmction may be used for MT system development. Introduction to the MT Plateau We believe it is fair to say that the field of machine translation has been on a plateau for at least the past decade. 2 Traditional, band-built MT systems held up very well in the ARPA MT evaluation (White and O'Connell 1994). These systems are relatively expensive to build and generally require a trained staff working for several years to produce a mature system. This is the current commercial state of the art: hand-building specialized lexicons and translation rules. A completely different type of system was competitive in this evaluation, namely, the purely statistical CANDIDE system built at IBM. It was generally felt that this system had also reached a plateau in that more data and more training was not likely to improve the quality of the output. Low Density Machine Translation However, in the case of "Low Density Machine Translation" (see Nirenburg and Raskin 1998, Jones and Havrilla 1998) commercial market forces are not likely to provide significant incentives for machine translation systems for Low Density (Non-Major) languages any time soon. Two noteworthy efforts to break past the data and labor bottlenecks for high-quality machine translation development are the following. The NSF Summer Workshop on i Douglas Jones is now at National Institute of Standards & Technology, Gaithersburg, MD 20899, Douglas.Jones @NIST.gov a A sensible, plateau-fi'iendly strategy may be to accumulate translation memory to improve both the long-term efficiency of human translators and the quality of machine translation systems. If we imagine that the plateau is really a kind of logarithmic function tending ever upwards, we need only be patient. Statistical Machine Translation held at Johns Hopkins University summer 1999 developed a public-domain version intended as a platform for further development of a CANDIDE-style MT system. Part of the goal here is to improve the trauslation by adding levels of linguistic analysis beyond the word N-gram. An effort addressing the labor bottleneck is the Expedition Project at New Mexico State University where a preliminary elicitation environlnent for a computational field linguistics system has been developed (the Boas interface; see Nirenburg and Raskin 1998) A Scoring Function for MT quality Our contribution toward working beyond this plateau is to look for a way to define a scoring function for the quality of the English output such that we can use it to machine-learn a good translation grammar. The novelty of our idea for this function is that we do not have to define the internals of it ourselves per se. We are able to define a successful function for two reasons. First, there is a growing body of software worldwide that has been designed to consume English; all we need is for each piece of software to provide a metric as to how Englishlike its input is. Second, we can tell whether the software had trouble with the input, either by system-internal diagnosis or by diagnosing the software's output. A good illustration is the facility in current word-processing software to put red squiggly lines underneath text it thinks should be revised. We know fi'om experience that this feature is often only annoying. Nevertheless, imagine that it is correct some percentage of the time, and that each piece of software we use for this purpose is correct solne percentage of the time. Our strategy is to